In modern computing systems, software quality assurance (SQA) is an important part of verifying software components before actual deployment of the components by end users. This generally includes a planned systematic pattern of the actions necessary to provide adequate confidence that a product, component, or process by which the product is developed, conforms to established requirements. Also, SQA methods are more broadly applied to any software development such as to an updated version of commercial software to correct errors, resolve incompatibilities, or improve software performance.
In accordance with SQA methodology, software reliability considerations include the probability that software will not cause a system failure for a specified time under specified conditions. This probability is generally a function of one or more inputs to and use of the system in accordance with the software. The inputs to the system determine whether existing faults, if any, are encountered. Moreover, reliability estimates include the ability of a program to perform its required functions accurately and reproducibly under stated conditions for a specified period of time. Furthermore, reliability includes the probability that a given software component operates for some time period on the machine or machines for which it was designed, without system failure due to a software fault, given that it is used within design limits of the software.
To achieve the above goals, various forms of testing are generally applied to a system to determine how a given system performs under load. For example, this may include testing in which a system is subjected to unrealistically harsh inputs or load with inadequate resources with the general intention of breaking or faulting the system. For instance, this can include testing conducted to evaluate a system or component at or beyond the limits of its specified requirements. Generally, stress tests are designed to confront programs with abnormal situations. Thus, stress testing executes a system in a manner that may demand system resources in abnormal quantity, frequency, or volume.
One important component of system/component stress testing relates to load testing that attempts to stress systems by exercising components beyond that which is perceived or determined to be worst case operating conditions for the system. For instance, companies with mission critical web applications cannot afford to have poorly performing web sites as demand for the sites changes. As web site growth evolves, and systems are upgraded, the complexity of a company's web site architecture increases, for example. Thus, as components within the web architecture change, applications operating the sites are increasingly likely to encounter performance issues. Consequently, software applications can experience a great number of users with unpredictable load variations.
Some companies offer load testing applications to measure and predict the behavior and performance of an application on a global scale. This may include load testing both a determined infrastructure and architecture of an application by simulating a large number of users with many different profiles. Also, these applications can be combined with performance and transaction monitors in order to provide specific performance data for the different components of the application architecture. An output of these load and stress tests can be reports that identify in real-time any bottleneck and its cause that has been experienced by the system.
One problem with conventional stress and load testing methodologies is that merely testing a system at its breaking point or overload condition may not be the most efficient way to determine reliability let alone determine whether or not the software under test will reliably operate as specified. Another problem with conventional testing methodologies relates to the rigidity in which testing scenarios are constructed and subsequently executed. For instance, when a new test is incorporated into a suite of tests, previous tests may have to be re-written or eliminated in order to allow the new test to execute properly. Other problems relate to the fact that it may not always be desirable to execute tests in a predetermined order nor may it be desirable to execute one type of test during the execution of another. For instance, it may be desirable to apply certain faults, loads, or stresses to a system to observe system behavior without also requiring the system under test to be monitored for a particular failure or any type of failure during the observation period. In order to achieve these and other type of testing conditions, many tests often have to be redesigned to enable such testing flexibility.